The present invention relates to an image-capturing apparatus for recording a wide dynamic range image data by image capturing, an image processing apparatus for applying optimization processing for forming an output-referred image on the outputting medium, to such a wide dynamic range image data by image capturing, and an image recording apparatus.
At present, the digital image data captured by an image-capturing apparatus is distributed through such a memory device as a CD-R (Compact Disk Recordable), floppy disk (registered trade name) and memory card or the Internet, and is displayed on such a display monitor as a CRT (Cathode Ray Tube), liquid crystal display and plasma display or a small-sized liquid crystal monitor display device of a cellular phone, or is printed out as a hard copy image using such an output device as a digital printer, inkjet printer and thermal printer. In this way, display and print methods have been diversified in recent years.
When digital image data is displayed and output for viewing, it is a common practice to provide various types of image processing typically represented by gradation adjustment, brightness adjustment, color balancing and enhancement of sharpness to ensure that a desired image quality is obtained on the display monitor used for viewing or on the hard copy.
In response to such varied display and printing methods, efforts have been made to improve the general versatility of digital image data captured by an image-capturing apparatus. As part of these efforts, an attempt has been made to standardize the color space represented by digital RGB (Red, Green and Blue) signals into the color space that does not depend on characteristics of an image-capturing apparatus. At present, large amounts of digital image data have adopted the sRGB (See Multimedia Systems and Equipment—Color Measurement and Management—Part 2-1: Color Management—Default RGB Color Space—sRGB” IEC61966-2-1) as a standardized color space. The color space of this sRGB has been established to meet the color reproduction area for a standard CRT display monitor.
Generally, a digital camera is equipped with an image sensor, serving as an image-capturing device (CCD type image sensor, hereinafter also referred to as “CCD” for simplicity) having a photoelectric conversion function with color sensitivity provided by a combination of a CCD (charge coupled device), a charge transfer device and a mosaic color filter. The digital image data output from the digital camera is obtained after the electric original signal gained by conversion via the CCD has been corrected by the photoelectric conversion function of the image sensor, and processing of file conversion and compression into the predetermined data format standardized to permit reading and display by image editing software.
Correction by the photoelectric conversion function of the image sensor includes, for example, gradation correction, spectral sensitivity, crosstalk correction, dark current noise control, sharpening, white balance adjustment and color saturation adjustment. The above-mentioned standardized data format widely known includes Baseline Tiff Rev. 6.0 RGB Full Color Image adopted as a non-compressed file of the Exif (Exchangeable Image File Format) file and compressed data file format conforming to the JPEG format.
The Exif file conforms to the above-mentioned sRGB, and the correction of the photoelectric conversion function of the above-mentioned image-capturing element is established so as to ensure the most suitable image quality on the display monitor conforming to the sRGB.
For example, if a digital camera has the function of writing into the header of the digital image data the tag information for display in the standard color space (hereinafter referred to as “monitor profile”) of the display monitor conforming to the sRGB signal, and accompanying information indicating the device dependent information such as the number of pixels, pixel arrangement and number of bits per pixel as meta-data as in the case of Exif format, and if only such a data format is adopted, then the tag information can be analyzed by the image edit software (e.g. Photoshop by Abode for displaying the above-mentioned digital image data on the digital display monitor, conversion of the monitor profile into the sRGB can be prompted, and modification can be processed automatically. This capability reduces the differences in apparatus characteristics among different displays, and permits viewing of the digital image data photographed by a digital camera under the optimum condition.
In addition to the above-mentioned information dependent on device type, the above-mentioned accompanying information includes; information directly related to the camera type (device type) such as a camera name and code number, information on photographing conditions such as exposure time, shutter speed, f-stop number (F number), ISO sensitivity, brightness value, subject distance range, light source, on/off status of a stroboscopic lamp, subject area, white balance, zoom scaling factor, subject configuration, photographing scene type, the amount of reflected light of the stroboscopic lamp source and color saturation for photographing, and tags (codes) for indicating the information related to a subject. The image editing software and output device have a function of reading the above-mentioned accompanying information and making the quality of hardware image more suitable.
The image displayed on such a display device as a CRT display monitor and the hard copy image printed by various printing devices have different color reproduction areas depending on the configuration of the phosphor or color material to be used. For example, the color reproduction area of the CRT display monitor corresponding to the sRGB standard space has a wide bright green and blue area. It contains the area that cannot be reproduced by the hard copy formed by a silver halide photographic printer, ink-jet printer and conventional printer. Conversely, the cyan area of the conventional printing or inkjet printing and the yellow area of the silver halide photographic printing contain the area that cannot be reproduced by the CRT display monitor corresponding to the sRGB standard color space. (For example, see “Fine imaging and digital photographing” edited by the Publishing Commission of the Japan Society of Electrophotography, Corona Publishing Co., P. 444). In the meantime, some of the scenes of the subject to be photographed may contain the color in the area that cannot be reproduced in any of these areas for color reproduction.
As described above, the color space (including the sRGB) optimized on the basis of display and printing by a specific device is accompanied by restrictions in the color gamut where recording is possible. So when recording the information picked up by a photographing device, it is necessary to make adjustment of mapping by compressing the information into the color gamut where recording is allowed. The simplest way is provided by clipping where the color chromaticity point outside the color gamut where recording is possible is mapped onto the boundary of the nearest color gamut. This causes the gradation outside the color gamut to be collapsed, and the image will give a sense of incompatibility to the viewer. To avoid this problem, non-liner compression method is generally used. In this method, the chromaticity point in the area where chroma is high in excess of an appropriate threshold value is compressed smoothly according to the size of the chroma. As a result, chroma is compressed and recording is carried out even at the chromaticity point inside the color gamut where recording is possible. (For the details of the procedure of mapping the color gamut, see “Fine imaging and digital photographing” edited by the Publishing Commission of the Japan Society of Electrophotography, Corona Publishing Co., P. 447, for example).
The image displayed on such a display device as a CRT display monitor, the hard copied image printed by various types of printing devices, or color space (including sRGB) optimized on the basis of display and printing by these devices are restricted to the conditions where the area of brightness that allows recording and reproduction is of the order of about 100 to 1. By contrast, however, the scene of the subject to be photographed has a wide area of brightness, and it often happens that the order of several thousands to 1 is reached outdoors. (See “Handbook on Science of Color, New Version, 2nd Print” by Japan Society for Science of Colors, Publishing Society of the University of Tokyo, P. 926, for example). Accordingly, when recording the information gained by the image sensor, compression is also necessary for brightness. In this processing, adequate conditions must be set for each image in conformity to the dynamic range of the scene to be photographed, and the range of brightness for the main subject in the scene to be photographed.
However, when compression has been carried out for the color gamut and brightness area as described above, information on gradation prior to compression or information prior to clipping is lost immediately due to the principle of the digital image to be recorded in terms of the discrete value. The original state cannot be recovered. This imposes a big restriction on the general versatility of high-quality digital image.
For example, when the image recorded in the sRGB standard color space is printed, mapping must be carried out again based on the differences between the sRGB standard color space and the area for color reproduction of the printing device. For the image recorded in the sRGB standard color space, however, the information on gradation in the area compressed at a time of recording is lost. So the smoothness of gradation is deteriorated as compared to the case where the information captured by the photographing device is mapped directly in the area for color reproduction of the printing device. Further, if gradation compression conditions are not adequate at a time of recording, and there are problems such as a whitish picture, dark face, deformed shadow and conspicuous white skipping in the highlighted area, improvement is very inadequate as compared to the case where the new image is created again from the information gained by the photographing device, even if the gradation setting is changed to improve the image. This is because information on gradation prior to compression, and information on the portion subjected to deformation or white skipping have been already lost.
As a solution of the above-mentioned problems, the art of storing the process of image editing as a backup data and returning it to the state prior to editing whenever required has long been known. For example, Patent Document 1 discloses a backup device wherein, when the digital image is subjected to local modification by image processing, the image data on the difference between the digital image data before image processing and that after image processing is saved as backup data. Further, Patent Document 2 discloses a method for recovering the digital image data before editing, by saving the image data on the difference between the digital image data before image processing and that after image processing. These technologies are effective from the viewpoint of preventing information from being lost, but the number of sheets that can be photographed by a camera is reduced with the increase in the amount of data recorded in the media.
The problems introduced above are caused by the procedure where the information on the wide color gamut and brightness area gained by a photographing device is recorded after having being compressed into the output-referred image data in the state optimized by assuming an image to be viewed. By contrast, if the information on the wide color gamut and brightness area gained by a photographing device is recorded as scene-referred image data that is not compressed, then inadvertent loss of information can be prevented. The standard color space suited to record such scene-referred image data is proposed, for example, by RIMM RGB (Reference Input Medium Metric RGB) and ERIMM RGB (Extended Reference Input Medium Metric RGB). (See the Journal of Imaging Science and Technology, Vol. 45 pp. 418 to 426 (2001)).
However, the data expressed in the standard color space like the one described above, is not suitable for being displayed directly on the display monitor and viewed. Generally, a digital camera has a built-in display monitor or is connected to it in order for the user to check the angle of view before photographing or to check the photographed image after photographing. When photographed data is recorded as output referred image data like the sRGB, it can be displayed directly on the display monitor, without the data being converted. Despite this advantage, when the photographed data is recorded as scene-referred image data, the data must be subjected to the processing of re-conversion into the output-referred image data in order to display that data.
Patent Document 3 discloses an image processing apparatus characterized by two modes; a mode of recording in the form of an image signal displayed on the display means and a mode of recording in the form of captured image signal. The form of image signal in the latter case is generally called RAW data. Using the special-purpose application software (called “development software”), such digital image data can be converted into output-referred image data of the above-mentioned Exif file or the like for display or printing (called “electronic development” or simply “development”). Since the RAW data retains all information at a time of photographing, it permits output-referred image data to be remade. If other color system files such as CMYK are created directly, there will no inadvertent modification of the color system due to the difference in color gamut from the display monitor (sRGB). However, the RAW data is recorded according to the color space based on the spectral sensitivity characteristics inherent to the type of a photographing apparatus and the file format inherent to the type of a photographing apparatus. Accordingly, image suitable to display and printing can be obtained only when special-purpose development software inherent to the type of the photographing apparatus is used.
The above description is concerned with a commonly used digital camera. Needless to say, it is preferred that the image-capturing apparatus itself be improved so as to get information on a wider color and brightness range than the conventional products. The solid image sensor has a limited dynamic range with respect to the intensity of incoming light. The Patent Document 4 discloses a method wherein a low-sensitivity image sensor and high-sensitivity image sensor are provided, and the value obtained by level conversion of the output of each of these devices is compared with a reference voltage; thus, the dynamic range of the signal obtained by photoelectric conversion of the intensity of incoming light is expanded by instantaneous switching of the output of either of the level values.
The Patent Document 5 discloses a technique wherein the first light receiving devices (high-sensitivity image sensor) and second light receiving devices (low-sensitivity image sensor) of the image sensor are arranged in a honeycomb figure in positions displaced, in the direction of vertical column and/or and horizontal row with respect to each other in terms of the centers of the geometrical profile of the image sensor, by half the pitch representing the interval between light receiving devices, namely, by ½ pitch; and the signal saturation level of the first light receiving device is adjusted for synthesization between the first and second signal.
Patent Document 6 discloses the method wherein the image sensor generates the high-sensitivity video signal and low-sensitivity video signal, and addition is made in terms of sensitivity ratio, if the high quantization data is saturated subsequent to quantization of the high-sensitivity video signal by high quantizing resolution and low-sensitivity video signal by low quantizing resolution, and the high quantization data is selected otherwise, thereby forming a wider dynamic range image.
Further, the Patent Document 7 describes the art of selecting the output values of the low-sensitivity image sensor and high-sensitivity image sensor according to the exposure area, and Patent Document 8 shows the method for selecting the output value of the low-sensitivity image sensor and high-sensitivity image sensor without moiré.
[Patent Document 1]                Tokkaihei 7-57074        
[Patent Document 2]                Tokkai 2001-94778        
[Patent Document 3]                Tokkaihei 11-261933        
[Patent Document 4]                Tokkohei 8-34558        
[Patent Document 5]                Tokkai 2000-125209        
[Patent Document 6]                Tokkai 2001-8104        
[Patent Document 7]                Tokkai 2003-18445        
[Patent Document 8]                Tokkai 2003-18479        
In a digital camera having the image-capturing section obtained by high integration of image sensors with different characteristics, loss of information can be avoided by referring to the information on wide color range and brightness as a non-compressed scene-referred image data (e.g. [scRGB], [RIMM RGB] and [ERIMM RGB]). However, the abovementioned technology has problems in that it fails to enable viewing by display on the aforementioned camera display monitor, and to overcome the variations of the image quality due to the camera model inherent problem, similarly to the case of raw data or the difficulty in optimization of the image quality for use in display and printing. This has created a problem in the prior art.